Light Field Displays Incorporating Eye Trackers and Methods for Generating Views for a Light Field Display Using Eye Tracking Information

ABSTRACT

A multiview autostereoscopic display includes a display area including an array of angular pixels, an eye tracker, and a processing system. Each angular pixel emits color that varies across a field of view of that angular pixel. The array of angular pixels displays different views in different viewing zones across the field of view of the display. The eye tracker detects the presence of the eyes of at least one viewer within specific viewing zones and produces eye tracking information including locations of the detected eyes within the specific viewing zones. The processing system renders a specific view for each detected eye based upon the location of the detected eye within the viewing zone with detected eyes, and generates control information for the array of angular pixels to cause the specific view for each detected eye to be displayed in the viewing zone in which that eye was detected.

PRIORITY CLAIM

The present application claims the benefit of copending U.S. PatentApplication Ser. No. 62/929,666, filed Nov. 1, 2019 and entitled “LightField Displays Incorporating Eye Trackers and Methods for GeneratingViews for Multiview Autostereoscopic Display Using Eye TrackingInformation,” which is incorporated herein by reference in its entirety.

FIELD OF THE INVENTION

The present invention relates generally to three-dimensional (3D)displays and more specifically to multiview autostereoscopic 3D displaysthat incorporate eye trackers.

BACKGROUND

The human vision system is able to interpret 3D structure from 2Dprojections using cues such as occlusion, perspective, familiar size,and atmospheric haze. However, 2D images are unable to represent anumber of significant depth cues including stereo parallax, motionparallax, accommodation, and convergence. 3D displays that provide adifferent image to each eye of a viewer are referred to as stereoscopicdisplays. Most commercially available 3D display technologies requirethe viewer to wear special glasses that allow the display to presentdifferent images to each eye of the wearer. A limitation of 3D displaysthat utilize special glasses to provide different images to each eye ofthe wearer is that the images are typically not dependent upon the headposition of the wearer. As conceptually illustrated in FIG. 1, a display100 presents a Left (L) and a Right (R) image to the left and right eyesof the viewer, respectively, ideally located at the ideal viewinglocation 102. These images will be the same irrespective of the locationof the viewer and the depth cues may become inconsistent with thecontent of the scene the further the viewer is located from the idealviewing location, such as at locations 104 and 106.

An alternative to using special glasses is to mount two small displaysin front of a viewer's eyes, such as within a pair of goggles or aheadset. As the viewer's head moves and the head motion is tracked bythe headset (e.g., using an accelerometer, a gyroscope, positionsensors, infrared emission, a camera, and/or a magnetometer), thedisplay can present viewpoint-dependent images on the two displays toprovide stereo parallax, and motion parallax. The need to wear a headsetand the resulting isolation from the real world can significantly limitthe usefulness of head-worn 3D displays such as virtual reality, mixedreality, and/or augmented reality glasses.

Autostereoscopic displays create a binocular perception of 3D depthwithout the requirement that the user wear special headgear, such asglasses or a headset. Autostereoscopic displays typically utilizespatial multiplexing of images, which can be implemented in a variety ofways including placement of parallax barriers in front of a display,placement of a lenticular lens array or microlens array in front of adisplay, and using an array of projectors.

Multiview autostereoscopic light field displays are a class ofautostereoscopic display that forms different images in differentviewing zones across the field of view of the display. As a result,multiview autostereoscopic light field displays are not limited tosimply providing different images to each of the viewer's eyes at agiven time, but as the user moves, the images displayed to each eye varyin a manner that is dependent upon the location of the viewer. In thisway, multiview autostereoscopic light field displays can provide bothstereo and motion parallax. However, commercialization of multiviewautostereoscopic light field displays has proven challenging due to thedifficulty of building a display that can display many views and thecomputational complexity of generating all the views simultaneously.

SUMMARY OF THE DISCLOSURE

The following presents a simplified summary of one or more aspects inorder to provide a basic understanding of such aspects. The summary isnot an extensive overview of all contemplated aspects, and is intendedto neither identify key or critical elements of all aspects nordelineate the scope of any or all aspects. The purpose is to presentsome concepts of one or more aspects in a simplified form as a preludeto the more detailed description presented later.

In an aspect, a multiview autostereoscopic display for providing aplurality of views to a viewer is presented. The display includes adisplay area, including an array of angular pixels. Each angular pixelis configured for emitting light that varies across a field of view ofthat angular pixel. The array of angular pixels is configured fordisplaying different views in different viewing zones across a field ofview of the display. The display further includes at least one eyetracker configured for detecting the presence of eyes of the viewerwithin specific viewing zones and producing eye tracking informationrelated to locations of the eyes so detected within the specific viewingzones. The display additionally includes a processing system configuredfor rendering a specific view for each one of the eyes so detected,based upon the location of that eye within the viewing zone in whichthat eye was detected, and generating control information for the arrayof angular pixels to cause the specific view for that eye to bedisplayed in the viewing zone in which that eye was detected.

In another aspect, processing system of the multiview autostereoscopicdisplay is further configured for rendering at least one of reducedresolution views, reduced density views, and two-dimensional views fordisplay in specific viewing zones based upon eye tracking information.

In still another aspect, the eye tracking information includesconfidence information indicating a level of uncertainty associated witha tracked eye location, and the processing system is further configuredfor adjusting the rendering according to the confidence information.

In yet another aspect, the processing system is configured forgenerating control information for a plurality of logical emitters. Thedisplay area further includes backplane circuitry configured forreceiving the control information for the plurality of logical emitters,and interpolating the control information for the plurality of logicalemitters to control information for the array of angular pixels in thedisplay area.

In another aspect, the display is configured for providing viewsviewable by a first viewer and a second viewer. The at least one eyetracker is further configured for simultaneously tracking eyes of thefirst and second viewers. The processing system is further configuredfor rendering specific views based upon a first scene for each detectedeye of the first viewer, and rendering additional views based upon asecond scene for each detected eye of the second viewer, the secondscene being at least partially different from the first scene. Theprocessing system is also configured for generating control informationfor the array of angular pixels such that, simultaneously, the specificviews for each detected eye of the first viewer are displayed in viewingzones in which the eyes of the first viewer were detected, and theadditional views for each detected eye of the second viewer aredisplayed in viewing zones in which the second viewer's eyes weredetected. In a further aspect, the control information so generated bythe processing system causes no view to be displayed outside the viewingzones in which at least one of the first and second viewer's eyes weredetected. In still a further aspect, the display includes a first eyetracker for tracking the eyes of the first viewer, and a second eyetracker for tracking eyes of the second viewer.

In another aspect, the processing system further includes memoryconfigured for storing calibration information for the array of angularpixels. The processing system generates the control information for theplurality of emitters in the angular pixels in the display area basedupon the calibration in formation and target intensities for a set ofray directions from the display area.

In yet another aspect, the processing system is further configured foroptimizing the control information for mitigating at least one ofghosting, optical crosstalk between viewing zones, and pixel-leveloptical crosstalk.

BRIEF DESCRIPTION OF THE DRAWINGS

The appended drawings illustrate only some of the consideredimplementations and are therefore not to be considered limiting ofscope.

FIG. 1 conceptually illustrates a 3D display that utilize glasses toprovide different images to each eye of the wearer.

FIG. 2 illustrates a light field display that includes a display areaformed by an array of angular pixels and one or more eye trackers, inaccordance with an embodiment of the invention.

FIG. 3 conceptually illustrates a high-level overview of processingsystems utilized within a personal computer incorporated with a lightfield display, in accordance with an embodiment of the invention.

FIG. 4 illustrates a light field display including angular pixels formedby lens elements and arrays of emitters, in accordance with anembodiment of the invention.

FIG. 5 illustrates a light field display implemented using projectors,in accordance with an embodiment of the invention.

FIG. 6A illustrates a hierarchical eye tracking process.

FIG. 6B illustrates a hierarchical gaze tracking process.

FIG. 7 conceptually illustrates a cone of light generated by an emitterin a light field display, in accordance with an embodiment of theinvention.

FIG. 8 conceptually illustrates a manner in which aliasing can result ina viewer perceiving background discontinuities in views produced by anautostereoscopic display.

FIG. 9 conceptually illustrates display of views rendered based ontracked locations of a viewer's eye within a viewing zone of anautostereoscopic light field display, in accordance with an embodimentof the invention.

FIG. 10 illustrates a process of operating a light field display tomodify the views displayed in viewing zones occupied by the eyes of aviewer based upon the tracked locations of the eyes of the viewer, inaccordance with an embodiment of the invention.

FIG. 11 conceptually illustrates the phenomenon of ghosting.

FIG. 12 conceptually illustrates a process that can be utilized by alight field display to render the images displayed in viewing zonesbased upon eye tracking information to reduce ghosting, in accordancewith an embodiment of the invention.

FIG. 13 illustrates a manner in which uncertainty in eye tracking can beaccommodated by displaying the same color and intensity in viewing zonesadjacent to tracked eyes, in accordance with an embodiment of theinvention.

FIG. 14 illustrates a representation of uncertainty in gaze directionand eye position as a 5D probability distribution.

FIG. 15 conceptually illustrates a process for performing foveatedrendering with a light field display, based upon eye and gaze trackinginformation in a manner that accounts for uncertainty in eye and gazetracking information, in accordance with an embodiment of the invention.

FIG. 16 conceptually illustrates a process for controlling theactivation of emitters within a light field display in accordance withan embodiment of the invention.

FIG. 17 conceptually illustrates a process that enables display of viewsin viewing zones adjacent to the viewing zones occupied by tracked eyes,in accordance with an embodiment of the invention.

FIG. 18 conceptually illustrates a light field display, in accordancewith an embodiment of the invention, displaying simplified content inviewing zones that are not occupied by tracked eyes.

FIG. 19A conceptually illustrates a manner in which intra-microlenscrosstalk can manifest.

FIG. 19B conceptually illustrates a manner in which inter-microlenscrosstalk can manifest.

FIG. 20 illustrates a process for controlling activation of emitters ofa light field display, based upon eye tracking information, in order todisplay views in viewing zones occupied by tracked eyes, in accordancewith an embodiment of the invention.

FIG. 21 conceptually illustrates a view rendering pipeline for a lightfield display incorporating eye tracking, in accordance with anembodiment of the invention.

DETAILED DESCRIPTION

Turning now to the drawings, light field displays that incorporate eyetrackers and processes that utilize eye tracking data in the generationof views for multiview autostereoscopic display in accordance withvarious embodiments of the invention are illustrated. The term lightfield display is used herein to refer to a class of multiviewautostereoscopic display that includes several orders of magnitude moreemitters than conventional autostereoscopic displays. Accordingly, lightfield displays achieve a significantly higher density of different viewscompared to conventional autostereoscopic displays. The increased numberof views comes with a potential proportionate increase in the processingrequired to generate the number of views utilized for multiviewautostereoscopic display.

Systems and methods in accordance with a number of embodiments of theinvention utilize eye trackers to reduce the number of views that aregenerated for autostereoscopic display. In many embodiments, views aregenerated corresponding to the specific locations of tracked eyes. Theseviews can be displayed within viewing zones occupied by the tracked eyesand, in many instances, adjacent viewing zones to accommodate eyetracking uncertainty. In many embodiments, the views are rendered fordisplay in overlapping viewing zones so that the superposition of theviews on a viewer's retina create a desired image. The integration ofeye tracker information with the data related to the generation of viewsfor a light field display provides a variety of advantages for reducingthe computational and energy loads required in providing light fieldviews by such a multiview autostereoscopic display.

An eye tracker is a system that can track the location of the eyes ofone or more people. An eye tracker typically determines the location ofeach viewer's eyes in three-dimensional space and, in many instances,determines the rotation of each eye with respect to a frame ofreference, yielding the gaze direction. In a number of embodiments, theeye tracker tracks the center of the pupil of each eye. In manyembodiments of the invention, the eye tracker determines the locationsof the centers of the focal plane of each eye. For instance, since thefocal plane of the eye is approximately opposite the pupil on the retinaand so the location of the focal plane can be inferred from a knownpupil location. In a number of embodiments, a hierarchical eye trackingprocess is utilized. An eye tracker can be differentiated from a headtracker, which simply determines the location of a viewer's head. An eyetracker can also be differentiated from a gaze tracker, which determinesthe location of the viewer's eyes and the direction in which the vieweris looking. While much of the discussion that follows relates to lightfield displays that incorporate at least one eye tracker and utilize eyetracking information to generate views for multiview autostereoscopicdisplay, systems and methods in accordance with several embodiments ofthe invention can utilize gaze trackers and/or head trackers as analternative to or in addition to eye trackers, as appropriate due to therequirements of specific applications. Furthermore, light field displaysin accordance with many embodiments of the invention can be utilized inhead mounted displays in which eye and/or gaze tracking is performed bythe head mounted display in conjunction with head pose tracking usingsensors such as (but not limited to) accelerometers within the headmounted display.

As mentioned above and is discussed further below, the ability toidentify viewing zones in which eyes are located enables light fielddisplays in accordance with many embodiments of the invention to achieveenergy, processing and/or design efficiencies. In several embodiments, alight field display can turn off emitters based upon eye trackinginformation to limit the views displayed to the viewing zones whereviewers are likely to be located. In this way, light field displays canachieve significant energy savings. Furthermore, bandwidth requirementsbetween the display's processing system and backplane control circuitrycan be reduced by only providing emitter control data for activeemitters and/or with respect to a set of logical emitters that issignificantly smaller than the actual number of physical emitters. In anumber of embodiments, the light field display can achieve additionalprocessing efficiencies by varying the resolution and/or extent to whichdisparity is rendered in views displayed within viewing locations inwhich no tracked eyes are detected (e.g., viewing zones adjacent aviewing zone in which a tracked eye is detected). Where gaze trackinginformation is available to a light field display, additional processingefficiencies can be achieved by rendering foveated views in whichresolution and/or disparity information within the rendered views variesbased upon gaze direction. Light field displays that incorporateparallel processing systems in accordance with many embodiments of theinvention can also employ advanced rendering pipelines that apply 3Dand/or 2D warps and translations post-render to continuously update theviewpoint from which a view is rendered based upon eye trackinginformation updates received during the view processing. In this way,the viewpoints from which displayed views are rendered can correspond asclosely as possible to the locations of each viewer's eyes at the pointin time that the light field display is refreshed and the views aredisplayed.

The ability to render different views within different viewing zonesbased upon eye tracking information also enables light field displays inaccordance with several embodiments of the invention to support avariety of useful modes. In many embodiments, the light field displaycan render views of scenes that are at least partially different todifferent viewers. In this context, a scene refers to a 3D model or apiece of content presented for viewing a user. In this way, two or moreviewers can simultaneously view two or more different pieces of contentvia a light field display in accordance with various embodiments of theinvention. In a number of embodiments, the light field display cansupport a privacy mode in which the viewing zones in which views aredisplayed are determined based upon the viewing zones occupied by theeyes of authorized viewers. The light field display can eitherdeactivate emitters that display views in viewing zones in which theeyes of unauthorized viewers are detected or present a view containing aprivacy warning or innocuous content.

In a number of embodiments, the use of an eye tracker enables the lightfield display to employ a hybrid approach for displaying horizontalparallax and vertical parallax. Instead of providing emitters to handlevertical parallax, eye tracking can be utilized to determine thevertical position and the distance of a viewer's eyes relative to thedisplay area. Once the vertical position and the distance of theviewer's eyes with respect to the display is known, the viewer'sperspective can be utilized to generate the view displayed in theviewing zones in which the viewer's eyes are located. In this way theeye tracker can reduce the total number of emitters within a light fielddisplay, because each angular pixel can be implemented with a single rowof emitters to handle horizontal parallax and vertical parallax ishandled computationally. In the absence of eye tracking, multiple rowsof emitters can be used to handle vertical parallax.

In several embodiments, the angular pixels utilized within the lightfield displays incorporate microlenses effecting optical aberrationsthat can result in light from emitters within the angular pixels leakinginto ray directions other than a primary intended direction. Forexample, the leakage of light from one emitter in an angular pixel intomultiple undesired ray directions is a form of optical cross-talk. In anumber of embodiments, groups of emitters within an angular pixel can bejointly controlled to achieve a desired linear combination of emittedintensities of all emitters. In many embodiments, the intensities of theemitters within an angular pixel are determined so that rays from theangular pixel perceived within one or more specific viewing zones are asclose as possible to the intended rays. In several embodiments, thelight emission of emitters from multiple angular pixels are determinedso that rays from the light field display perceived within one or morespecific viewing zones are as close as possible to the intended rays. Incertain embodiments, eye tracking information can be utilized todetermine the specific emitters contributing to views displayed withinviewing zones occupied by tracked eyes. For example, when a user isfurther from the light field display emitters a greater number ofangular pixels covering a larger area of the light field display maycontribute to a perceived ray. Similarly, when the perceived ray is inthe fovea (i.e., the center of the field of vision) of a user's field ofview, then the angular pixels that contribute to the perceived rayrepresent a smaller area of the light field display than when theperceived ray is in the periphery of the user's field of view. In manyembodiments, the intensities of the jointly controlled emitters aredetermined to minimize the difference between the perceived light fieldand a target light field. In several embodiments, however, additionalconstraints are imposed upon the jointly controlled emitters including(but not limited to) constraints that increase the dynamic range and/orblack level of the light field display. As can readily be appreciated,the specific constraints utilized to jointly control emitters in one ormore angular pixels within light field displays in accordance withvarious embodiments of the invention are largely dependent upon therequirements of a given application.

While much of the discussion herein is in the context of light fielddisplays, processes in accordance with various embodiments of theinvention can be utilized in any multiview autostereoscopic displaysystem including (but not limited to) parallax barrier, lenticular,microlens, holographic, head mounted, and/or projector-based multiviewautostereoscopic display systems that incorporate one or more eyetrackers as appropriate to the requirements of a given application.Light field displays that incorporate eye trackers and processes forutilizing eye tracking data to generate images for multiviewautostereoscopic display in accordance with a number of embodiments ofthe invention are discussed further below.

Light Field Displays Incorporating Eye Trackers

A light field display 200 that includes a display area 202 formed by anarray of angular pixels and one or more eye trackers 204 in accordancewith an embodiment of the invention is illustrated in FIG. 2. Eachangular pixel can be thought of as similar to a conventional pixel in a2D display with the difference that its appearance can vary across thefield of view of the display. In this way, each viewing zone of thelight field display can display an image with a resolution equal to thenumber of angular pixels in the light field display. Each angular pixelcan include an array of light emitters, such as described for example inUS Pat. Pub. No. US2019/0333443 A1, entitled “Architecture for LightEmitting Elements in a Light Field Display,” US Pat. Pub. No.US2019/0335165 A1, entitled “Partial Light Field Display,” and US Pat.Pub. No. US2019/0333444 A1, entitled “Architecture for Light EmittingElements in a Light Field Display,” all of which applications areincorporated herein by reference in their entirety. As is discussedfurther below, the number of emitters used to implement each angularpixel can be several orders of magnitude higher than the number ofangular pixels.

The manner in which views are generated for autostereoscopic display bya light field display is largely dependent upon the particularapplication in which the light field display is being utilized. Ahigh-level overview of a processing system utilized within a personalcomputer that incorporates a light field display in accordance with anembodiment of the invention is conceptually illustrated in FIG. 3. Aprocessing system 300 includes an eye tracking system 301. In anembodiment, eye tracking system 301 includes one or more image sensors302, such as a digital sensor array, infrared sensor, and/or a depthsensor. For instance, image sensors 302 may capture informationregarding a viewer of the display for use in eye tracking processes.Additionally, image sensors 302 may capture live images and/or video tobe displayed by a light field display. Eye tracking system 301optionally includes a dedicated embedded system 304, which may beintegrated into eye tracking system 301 as a dedicated unit forprocessing image input from image sensors 302 and generating eyetracking information. The specific processes that may be used ingenerating the eye tracking information are described in more detailbelow. Alternatively, image sensors 302 provide image data directly to adisplay stream processor 306, which drives an angular pixel array 308.Display stream processor 306 may process the image input from imagesensors 302 to generate the eye and/or gaze tracking information, or maypass on the information to a host personal computer (PC) 310 to generatethe eye tracking information. Exchange of data between image sensors302, optional, dedicated embedded system 304, display stream processor306, angular pixel array 308, and host personal computer (PC) 310 maytake place via any of a variety of connections including (but notlimited to) MIPI, I2C, USB, display port, video, HDMI, PCIE, and/orother connections. In the illustrated embodiment, rendering of views fordisplay by angular pixel array 308 is performed using a processingsystem with a host personal computer (PC) 310 such as (but not limitedto) an application processor (e.g., central processing unit (CPU)) and agraphics processing unit (GPU). The host PC 310 can receive eye and/orgaze tracking information from dedicated embedded system 304 via thedisplay stream processor 306 and can utilize the information to renderspecific views that can then be utilized by the display stream processorto provide control information to a backplane of angular pixel array308, which is utilized to generate the light field views.

Although specific hardware configurations are described above withrespect to FIG. 3, any of a variety of dedicated and/or softwareconfigured processing hardware can be utilized to obtain eye trackinginformation and to render views used to control a light field displaybased upon eye tracking information as appropriate to the requirementsof specific applications in accordance with various embodiments of theinvention. Furthermore, the specific hardware components and/or thelocations of the hardware components will typically vary depending uponthe use case for the light field display (e.g., personal computer,gaming computer, television display, advertising billboard, etc.) andthe techniques utilized to render the light field display.

In a number of embodiments, each angular pixel is formed by an array ofemitters positioned beneath each lens element in a lens array. A lightfield display 400 including angular pixels formed by lens elements andarrays of emitters is illustrated in FIG. 4. As noted above, aconventional multiview autostereoscopic display typically has acomparatively small number of emitters per lens element (i.e., 10s ofemitters at most). Light field displays in accordance with a number ofembodiments of the invention can include 100s or 1000s of emitters. Forinstance, each emitter array as shown in FIG. 4 may include a hundred ormore emitters, each emitter contributing to at least one of multipleviews, such as one of views A and B.

In another embodiment, a light field display 500 of FIG. 5 utilizes arear projection architecture to provide a multitude of light fieldviews. Light field display 500 includes multiple emitter arrays and alens array, in a manner similar to light field display 400 of FIG. 4.However, rather than providing two distinct stereoscopic views A and B,as in FIG. 4, the emitter arrays and lens arrays are configured tocooperate to provide multiple views at each point on a projectionscreen, as represented by a fan of views viewable by viewers atdifferent locations beyond the projection screen. Thus, viewers locatedat different locations see different views from light field display 500.

While much of the discussion that follows refers to light field displaysimplemented using lens elements and arrays of emitters as shown in FIG.4, light field displays in accordance with various embodiments of theinvention can also be implemented using projectors in a manner similarto the light field display 500 shown in FIG. 5. In addition, light fielddisplays can be implemented via direct projection using an array ofprojectors. While the use of light field displays is desirable due tothe density of views that can be produced by the light field display,systems and methods in accordance with various embodiments of theinvention are in no way limited to the use of light field displays.Accordingly, systems and methods in accordance with a number ofembodiments of the invention are implemented using a variety ofconventional multiview autostereoscopic displays as appropriate to therequirements of specific applications.

Eye and Gaze Tracking Processes

As is discussed below, the manner in which a light field displaypresents different views in viewing zones across the field of view ofthe display is impacted by the confidence with which the eye trackingsystem is able to track the eyes of one or more viewer. In manyembodiments, the light field display is designed to gracefully degradethe quality of the images displayed in particular viewing zones basedupon the presence or absence of tracked eyes and/or the confidence withwhich a pair of eyes have been located. In several embodiments, multipleeye tracking systems and/or eye tracking modalities can be utilized toincrease the likelihood that the light field display is able to reliablytrack the location of a viewer's eyes. For instance, multiple eyetracking systems positioned with a known baseline can be utilized toimprove the reliability with which the 3D position of a viewer's eyes isestimated. Furthermore, providing eye trackers with different fields ofview and/or multiple eye trackers oriented at different angles canenable eye tracking to continue when one or more of a viewer's eyes areoccluded within the field of view of one of the eye trackers but notwithin the field of view of a second eye tracker.

Eye trackers utilized within light field displays in accordance withvarious embodiments of the invention can be implemented using any of avariety of commercially available eye tracking technologies to provideeye tracking information, which typically includes 3D locationinformation for each tracked eye and, in many instances, one or moreconfidence metrics reflecting confidence in the detection of a trackedeye and/or the accuracy of the location information for the tracked eye.As can be appreciated from the discussion below, the greater theaccuracy of the 3D eye tracking information provided by the eye tracker,the more useful the eye tracking information is to the light fielddisplay.

In a number of embodiments, an eye tracker is utilized that isimplemented using a sensor system that projects infrared light and acamera system that captures images of the scene that include an infraredchannel. The use of infrared light for eye tracking has the advantagethat the eye tracker works well in indoor and/or low-light environments.Depth estimation based upon projected infrared light typically does notwork well in the presence of natural sunlight. Therefore, eye trackersin accordance with several embodiments of the invention can also utilizemultiple cameras to track eyes using, for example, multiview techniques.In several embodiments, eye locations can be determined in multipleimages captured from different viewpoints and disparity between the eyelocations can be utilized to determine the distance to the eye locationsfrom one or more cameras. In this configuration, the cameras can bemonochrome and/or capture image data in multiple color channels. Inseveral embodiments, the camera system employed in the eye tracker iscapable of capturing image data in the visible and infrared portions ofthe spectrum. The use of multiple cameras (particularly more than twocameras) can improve accuracy, handle occlusions more robustly, and/orsupport eye tracking across a wider field of view. In severalembodiments, light-field camera based eye tracking is utilized. It isrecognized herein that, the specific eye tracking sensors utilized toobtain eye tracking information for use in a light field display islargely dependent upon the requirements of a given application. Themanner in which eye tracking information can be utilized to generateviews for multiview autostereoscopic display in accordance with variousembodiments of the invention is discussed further below.

Referring again to FIG. 2, light field displays can include an eyetracking system 204 positioned above the display area 202. Light fielddisplays in accordance with various embodiments of the invention caninclude one or more eye tracking systems located above, below, to theside of, or integrated within the display area 202. In severalembodiments, one or more eye tracking systems are provided that aredistinct units housed separately from the display area 202 formed by thearray of angular pixels and can be independently positioned.Alternatively, specific areas within the display area 202 may includesensors configured for providing eye tracking information.

In general, eye tracking can be a noisy process. For instance, eyetracking noise can introduce perspective noise and/or increase thecomplexity of determining the views to render using a light fielddisplay. A variety of filtering processes can be applied to eye trackingmeasurements to decrease errors in the instantaneous tracked location ofa given eye. However, application of filtering processes can (in someinstances) introduce a time delay between the instant at which sensordata is captured and the determination of a reliable tracked eyelocation from the sensor data. In addition, the filtering process canprevent the eye tracking process from reacting quickly to rapid eyemovements. As is discussed in more detail below, the quality of a lightfield display can be impacted by differences between the eye locationutilized to render a view and the actual location of the tracked eye atthe time in which the view is displayed. Therefore, the benefits ofusing a filtering process to increase the accuracy with which an eye istracked at a particular point in time diminish when the latency of theprocess means that the eye is no longer likely to be at the trackedlocation when the view is rendered for display. In addition, thesmoothing applied by a filtering process can actually introduce a sourceof error when a rapid eye movement occurs. The difference between thetracked location of an eye generated by an eye tracking process at aparticular point in time and the actual location of the eye at that timeis often referred to as the lag between the tracked position and theactual position. Increasing the rate at which eye tracking is performedcan be an effective method for reducing noise (i.e., errors) in trackedeye locations without introducing a significant time lag.

In a number of embodiments, tracking the head, the face, the facialfeatures, the eyes, the irises and the pupils all or part jointly orusing hierarchical tracking can increase the computational efficiency oftracking eyes at high sampling rates without significantly increasinglatency within the eye tracking system. In several embodiments, ahierarchical eye tracking process is utilized that involves detectingand tracking the presence of at least one viewer. In many instances,viewer tracking is performed in a manner that differentiates betweendifferent viewers. For each tracked viewer, the hierarchical eyetracking process can also detect and track the head and/or facialfeatures of each viewer. In certain embodiments, such as shown in FIG.6A, the head tracking process is able to determine the location and/ororientation of each viewer's head. Once a viewer's head is located,hierarchical eye tracking processes in accordance with many embodimentsof the invention can determine the viewer's inter-pupil distance (IPD).This IPD may be refined over successive frames/observations. Thehierarchical eye tracking process can also detect and track combinationsof a face, facial features, an eye pair, an eye, an iris, and/or a pupillocation and orientation. In several embodiments, the IPD can beutilized to constrain the search for viewer's eyes to increase thecomputational efficiency of the eye tracking process. In addition,filtering and/or prediction processes including (but not limited to)statistical filtering processes can also be utilized to predict thelocation of tracked pupil locations (potentially based upon the locationof another tracked eye and/or IPD) in the event that tracking for aparticular eye is lost for some period of time.

As can readily be appreciated, any of a variety of joint, partiallyjoint, and/or hierarchical tracking processes can be utilized to trackeyes as appropriate to the requirements of specific applicationsincluding (but not limited to) hierarchical gaze tracking processessimilar to the process conceptually illustrated in FIG. 6B. For example,like the hierarchical eye tracking process illustrated in FIG. 6A, thehierarchical gaze tracking process of FIG. 6B begins with identificationof the body location, then head location, face orientation, eyeorientation, then pupil orientation in order to determine the gazedirection. Furthermore, the specific eye and/or gaze tracking processthat is utilized is largely dependent upon the sensors available forperforming eye tracking, the processing power, and/or the accuracyrequirements of a specific application. The manner in which light fielddisplays in accordance with various embodiments of the invention canutilize eye tracking and/or gaze tracking information to render viewsfor display in different viewing zones is discussed further below.

Eye Position Dependent Rendering

Light field rendering was described in Levoy, M. and Hanrahan, P., 1996,August, “Light field rendering,” In Proceedings of the 23rd annualconference on Computer graphics and interactive techniques (pp. 31-42),ACM. This publication is hereby incorporated by reference in itsentirety. In light field rendering, a scene can be represented as afour-dimensional function over position and angle, assuming anon-participating medium. The light field can be parameterized as allpossible rays between a camera plane (u,v) and a focal plane (s,t). Foreach such ray the light field stores a color. In practice, a subset ofthe light field is often represented as an array of two dimensionalimages. The (s,t) plane can be quantized by the resolution of the imagesmaking up the light field, and the (u,v) plane can be quantized bycamera positions. Given an arbitrary eye position, a novel view of thelight field may be synthesized by sampling a ray L(u,v,s,t) for eachpixel of the output image. If the ray L(u,v,s,t) is not available, itmay be interpolated from nearby ray L(u′,v′,s′,t′) values that arestored in the light field. Since the number of rays required torepresent a scene grows as a power of four rather than two, light fieldrendering has historically been considered to be computationallydemanding.

Light field displays in accordance with many embodiments of theinvention can utilize light field rendering to render views for displayin particular viewing zones. This rendering may occur in software,and/or in hardware. In many embodiments, the rendering process hasaccess to eye tracking information that specifies a number of viewer eyelocations (x,y,z) relative to the display. In several embodiments, theeye tracking information also includes confidence information thatindicates the level of uncertainty associated with a particular trackedeye location. In a number of embodiments, a rendering process can beutilized that dedicates more processing resources to sampling and/orinterpolating rays from the light field L(u,v,s,t) to generate views fordisplay in viewing zones that are likely to be observed by a viewer'sretina, while rays that are unlikely to be visible to an observer mayinvolve fewer processing resources.

In certain embodiments, the computational resources dedicated to therendering of particular views can be modulated by controlling thedensity with which rays are sampled in angle and/or space. In addition,computational resources can be modified by controlling the resolutionwith which a particular view is rendered. In rasterization embodiments,the sampling of the (u,v) and/or (s,t) plane—that is, the spatialresolution of individual views, or the angular spacing between adjacentviews—may be modified to allocate more resources to rays that are likelyto be observed, and less resources to rays that are likely to fallbetween viewers. In raytraced embodiments, the density of rays sampledmay be adjusted to prioritize rays intersecting with the areas aroundtracked eye positions. Given the compute resources required to trace afixed number of rays, when the eye position has higher confidence, morerays passing through a smaller sphere of space may be traced. If the eyeposition is known only with lower precision, the sphere around thelikely eye position will expand and the same number of rays may bedistributed across a larger space. Accordingly, light field displays inaccordance with many embodiments of the invention can allocate a givenamount of total rendering capacity per frame based upon a set of trackedviewpoints and the confidence of the tracking information.

As is discussed further below, light field displays in accordance withmany embodiments of the invention can utilize tracked eye and/or gazelocations to reduce computational complexity and/or reduce renderingartifacts. In a number of embodiments, the goal of the rendering processis to utilize eye tracking information to render views so that viewersperceive correct motion parallax, perspective, stereo disparity (i.e.,vergence), and accommodation. In addition, the rendering process shouldlimit and/or eliminate ghosting between overlapping views and provide astable viewing experience that avoids views changing over time in theabsence of change in the viewer's eye locations. Rendering processes inaccordance with various embodiments of the invention are discussedbelow.

Eye Position Dependent Rendering

In a perfect light field display, each angular pixel would include aninfinite number of light emitters. While light field displays inaccordance with many embodiments of the invention can include an orderof magnitude more emitters than conventional multiview autostereoscopicdisplays, the pitch of the emitters is still non-zero and so theviewings zones are quantized across the field of view of the display andnot smoothly continuous. As such, each emitter corresponds to a cone oflight generated by the display as is conceptually illustrated in FIG. 7.A light cone 700 produced by emitter 702 corresponds to a cone ofdirectional light from a single pixel (e.g., including an array of lightemitters) contributing to an image viewed in a particular viewing zonein device space shown above a display plane in FIG. 7. A challenge facedby conventional multiview autostereoscopic displays is that the spatialresolution with which angular pixels represent a scene is depthdependent. In other words, the spatial resolution is higher at alocation closer to the display plane as compared to at a locationfarther from the display plane. As can be appreciated from FIG. 7, thelight cone 700 produced by the emitter 702 corresponds to a narrowerregion at a depth plane 704 closer to the display plane than a region ata depth plane 706 further from the display plane. It is noted that aplurality of emitters 702, such as in a two-dimensional array, areassumed to be disposed in the display plane to contribute to light fieldimages projected therefrom. In other words, the size of the smallestvisible detail is related to distance from the display plane such thatfiner details of an object located at depth plane 704 closer to thedisplay plane in content space would be visible than for an objectlocated at depth plane 706 further from the display plane. Therefore,objects in the foreground located within a depth budget of the display(i.e., near depth plane 704) can be rendered with finer detail thanobjects in the background (i.e., at depth plane 706). As a result,generation of images for multiview autostereoscopic display typicallyinvolves sampling a scene with a spatial frequency that decreases withincreased distance from the display plane.

Decreasing the spatial sampling frequency of a highly textured scene(e.g., a scene in which the background includes a great deal of finedetail) can result in aliasing, which can be detrimental to the viewingexperience. In the context of autostereoscopic displays, aliasing istypically perceived as discontinuities or jumps in the background of thescene as a viewer moves between adjacent viewing zones. The effect ofthese discontinuities is that viewers do not experience realistic motionparallax as their eyes move within and between viewing zones. Anexemplary manner in which aliasing results in a viewer perceivingbackground discontinuities is conceptually illustrated in FIG. 8.Different emitters in an angular pixel produce first and second lightcones 800 and 802, respectively, corresponding to the same pixellocation in images within adjacent, first and second viewing zones 804and 806, respectively. When the images that are produced in the adjacentviewing zones contain aliasing, the viewer will experience adiscontinuity in the viewed scene as their eye moves from the firstviewing zone 804 to the second viewing zone 806, even if the imagedscene remains stationary between the two view zones.

Aliasing can be somewhat addressed through pre-filtering, which removeshigh-frequency components of the scene likely to introduce aliasing.Application of a pre-filtering process can result in images displayed bya multiview autostereoscopic display having blurred backgrounds that areconsistent as the viewer moves between adjacent viewing zones. Theeffect of application of a depth-dependent pre-filter (often referred toas a depth-dependent anti-aliasing filter) to an image of an object inthe foreground with a detailed background results in a blurring of thedetail of the background as well as potentially a reduction in thedetail of the foreground object. Such loss of image fidelity due topre-filtering is often undesirable. However, systems and methodsdescribed herein can reduce effects of aliasing resulting from theviewer moving from one viewing zone to the next by adjusting thedisplayed image at adjacent viewing zones according to the location ofthe tracked eyes of the viewer.

In another example, systems and methods in accordance with a number ofembodiments of the invention can increase motion parallax accuracy byutilizing eye tracking data to modify the image perceived by a viewer asthe viewer moves within a viewing zone and between viewing zones. Theprocess is conceptually illustrated in FIG. 9. As the viewer's eye movesacross a viewing zone within the left light cone shown in FIG. 9, thelocation of the eye is tracked by an eye tracking mechanism integratedwith the light field display such that the scene is resampled from thenew viewpoint of the viewer's eye in motion. The light field display canthen modify the intensity of the directional light from the emitterarray within the light field display to display the resampled image ateach eye location. By resampling the scene as the viewer moves, thescene is effectively sampled at a higher spatial frequency than theangular resolution of the viewing zones of the display. In this way,similarly to the process above with to mitigate aliasing, instead ofabrupt transitions between viewing zones, near continuous resampling ofthe scene based upon the location of the viewer's eyes can result insmooth transitions between adjacent viewing zones. Resampling the sceneand updating the display accordingly enables the light field display toreduce the extent of or eliminate entirely the backgrounddiscontinuities that would otherwise result due to aliasing, evenwithout application of a depth-dependent anti-aliasing filter.

In addition to resampling the scene and updating the view that isrendered as a viewer moves within a viewing zone and even acrossadjacent viewing zones, aliasing artifacts can be further reduced bymanaging the depth at which various objects within a scene are rendered.In a number of embodiments, the depth information of a scene is modifiedduring rendering by the light field display so that most of the contentof the scene is located at the plane of the light field display. Inseveral embodiments, objects within the scene are flattened duringrendering so that they are positioned more closely together along anaxis extending in a direction from the center of the display to theviewer. In this way, the scene is flattened in a different manner duringrendering for each viewer depending on the viewer location. The effectof the flattening is to increase the sharpness of the rendered view andto decrease aliasing. When flattening is viewpoint dependent, theflattening process can preserve consistency between object placement asa viewer moves to mimic expected motion parallax. The extent to whichdepth information assigned to objects within a scene is modified duringthe rendering process is largely dependent upon the requirements of aspecific application.

Uncertainty in eye tracking location information can result in renderinga view for display within a viewing zone that is from a perspective thatis different from the actual viewpoint of one or both of the viewer'seyes. As a viewer moves, the viewer expects to observe motion parallax.The amount of the motion parallax observed is typically dependent uponthe amount of movement. Where an initial view is rendered based upon anerror in eye tracking location and a subsequent view is rendered in amanner that corrects for tracking error, the extent of the motionparallax from the initial view to the corrected view may appear greaterthan the viewer expects based upon the extent of their body movement.

In order to preserve realistic motion parallax, light field displays inaccordance with a number of embodiments of the invention rendersequences of views based upon relative motion between tracked eyelocations as opposed to absolute eye tracking locations. In this way,the views can be rendered in a manner that results in motion parallaxcorresponding to the extent of the viewer's movements. For example, whenthe viewer looks away from a tracked eye location, moves their eyesrapidly, and/or blinks, the light field display can update the trackedeye locations to eliminate accumulated tracking errors. In severalembodiments, the light field display can determine the perspective fromwhich to render a view in a manner that is dependent both on absolutetracked eye location and the extent of eye movement to correct forprevious errors in eye tracking over time. In many embodiments, thecorrect perspective is restored when the content itself is rapidlychanging over a sequence of frames. Any of a variety of processes thataccommodate rendering views from the actual tracked eye locations and/orpreserve the accuracy of the motion parallax experienced as the usermoves can be utilized as appropriate to the requirements of specificapplications in accordance with various embodiments of the invention.

Further, depth cues observed by a viewer include the extent of thestereo disparity between views rendered for each of a viewer's eyes. Theextent of the stereo disparity rendered between the views is typicallydependent upon the baseline distance between the viewer's eyes orinter-pupil distance (IPD). In a number of embodiments of the invention,tracking of a viewer's eyes is performed in a manner that imposes arequirement of near constant IPD for each viewer. In this way, trackinguncertainty can be constrained in a manner that enforces consistency ofstereo disparity depth cues between the views rendered for display inthe viewing zones occupied by each of a viewer's eyes even as theviewer's eyes are in motion.

Additionally, a sequence of views can be rendered based upon the trackedlocations of the viewer's eyes. In many embodiments, the manner in whicha time sequence of views is rendered can account for the extent ofmotion parallax expected to be observed by a viewer based upon theviewer's movements.

In some instances, a viewer's pupil may span multiple, potentiallyoverlapping viewing zones. In which case, light field displays inaccordance with many embodiments of the invention can render multipleeye/pupil position dependent views for display in the multiple viewingzones visible to a viewer's eye. When views are rendered in differentviewing zones visible to a viewer's eye in a manner that isinconsistent, the viewer can perceive ghosting. When the multiple viewsthat are visible to a specific eye are rendered in a way that causes allof the views to move in a manner that is consistent, the views canprovide accommodation depth cues. Accordingly, light field displays inaccordance with many embodiments of the invention support one or moremodes, in which the same view is displayed within overlapping viewingzones visible to an eye, or different views containing accommodationcues (e.g., rendered so that the angular distance between the views isthe same as the angular distance between the viewing zones) aredisplayed in overlapping viewing zones. When only a portion of a viewingzone intersects a viewer's pupil, the angular distance between thatviewing zone and the neighboring viewing zone can be determined from thecenter of the segment of the viewing zone visible to the pupil. As withthe scenarios described above, the rendering can also consider theinter-pupil distance and relative location of the tracked eyes topreserve the consistency between motion parallax depth cues and stereodepth cues present in the views rendered by the light field display.

A process that can be utilized by a light field display to modify theview displayed in viewing zones occupied by the eyes of a viewer basedupon the tracked locations of the eyes of the viewer in accordance withan embodiment of the invention is shown in FIG. 10. According to thetracked location of the viewer's eyes, the light field display'sprocessing system can render views of a scene for display in a mannerthat is viewpoint dependent. While process 1000 illustrated in FIG. 10is focused on eye tracking for a single viewer, process 1000 can beexpanded to multiple viewers, provided additional capabilities toseparately track the eyes of multiple viewers are available. The process1000 includes attempting to track the location of the eyes of a viewerin a step 1001. When a determination 1002 is made that the viewer's eyeshave been located with high confidence, process 1000 proceeds to adecision 1003 to determine whether both of the viewer's eyes are withinthe same viewing zone.

If decision 1003 determines that both of the viewer's eyes are withinthe same viewing zone, then the process can render a single, joint eyeposition-dependent view for both eyes in a step 1004. In a number ofembodiments, the joint eye position-dependent view can be rendered basedupon a mid-point between the two tracked eye locations (potentiallyconstrained by a viewer specific inter-pupil distance).

If decision 1003 determines that each eye of the viewer is locatedwithin a different viewing zone, then unique eye/pupil positiondependent views can be rendered for each eye by sampling 1006 the scenefrom the viewpoint of each of the viewer's eyes. In several embodiments,the sampling 1006 of the scene is constrained to restrict the angulardistance between the rendered views to correspond to a viewer-dependentinter-pupil distance to increase the consistency of the stereo disparitydepth cues perceived by the viewer with other depth cues including (butnot limited to) motion parallax depth cues.

Whether or not the viewer's eyes are located within the same viewingzone, the views rendered in step 1004 or 1006 can then be used tocontrol 1007 the activation of the emitters within the light fielddisplay that display viewpoint-dependent images within the appropriateviewing zone(s) occupied by each of the viewer's eyes. In this way, theviews displayed to the viewer are not only dependent upon the specificviewing zones occupied by the viewer's eyes but the tracked locations ofthe viewer's eyes within those viewing zones.

Returning to determination 1002, if neither of the viewer's eyes hasbeen located with high confidence, then sampling 1012 of the scene isrestricted to rendering for a conventional multiview autostereoscopicdisplay. In other words, a determination 1002 that no eye can bedetected and/or that the noise or uncertainty in tracked eye locationsis high can result in the light field display sampling the scene for aconventional multiview autostereoscopic display, in which differentviews are displayed in each viewing zone across the field of view of thelight field display in a manner in which the view in a particularviewing zone does not change based upon the location of a viewer's eyewithin the viewing zone. When eye tracking is uncertain, attempting torender views in a viewpoint dependent manner can make the perceivedexperience of a light field display worse than simply rendering theviews in the manner of a conventional multiview autostereoscopicdisplay. In several embodiments, the light field display can respond toan absence of a tracked eye within a viewing zone or a loss of reliableeye tracking by controlling the emitters to function in a manner similarto a conventional 2D display in which the same view is displayed in eachviewing zone that does not contain a tracked eye. Then, the emitters ofthe light field display are controlled 1014 to display the renderedviews from step 1012 in corresponding viewing zones. In a number ofembodiments, the light field display displays autostereoscopic viewsrendered with coarser spatial resolution and/or stereopsis in viewingzones that do not contain a tracked eye. Processes in accordance withvarious embodiments of the invention can render views in a manner thatis dependent upon the uncertainty with which the eyes of multipleviewers are tracked and/or the uncertainties introduced by potentiallyoverlapping viewing zones in non-ideal light field displays arediscussed further below.

The process of tracking 1001 the viewer's eyes, rendering 1004 or 1006or 1012 the scene, and controlling 1007 or 1014 the display can berepeated until a determination 1008 is made that the process is complete(e.g., a user instruction deactivating the display or no user detectedfor user instructions received for a period of time).

While the discussion above primarily focuses on the display ofautostereoscopic views in viewing zones occupied by tracked eyes of asingle viewer, the ability to track the eyes of multiple viewers andrender views specific to the viewing zones occupied by each viewer'stracked eyes enables light field displays in accordance with manyembodiments of the invention to display different scenes to differentviewers. For example, multiple viewers could play a multiplayer videogame and each viewer of the light field display will perceive adifferent scene. The circumstances in which it can be beneficial for alight field display to render and display different scenes for differentviewers are not limited to gaming, and a decision concerning thedisplays to present distinct content to individual viewers is onlylimited by the requirements of specific applications, in accordance withvarious embodiments of the invention. Alternatively, light fielddisplays in accordance with several embodiments of the invention cansupport a mode in which each viewer sees a preferred viewpoint. Theability to display the same viewpoint to all viewers may be useful forapplications including (but not limited to) training presentationsand/or display of legacy stereo content for which insufficient depthinformation is available for full light field display.

When eye trackers are unable to track a viewer's eyes (e.g., no vieweris detected and/or the eye tracker loses tracking on one or both of theviewer's eyes), light field displays in accordance with many embodimentsof the invention can fall back to a number of different modes asappropriate to the requirements of specific applications.

Reducing Optical Crosstalk Between Viewing Zones

Due to manufacturing tolerances, optical components utilized withinlight field displays, such as microlenses used to implement angularpixels, can include aberrations that can cause light from an emitter toleak into ray directions other than the primary intended direction ofthe light emission. Leakage of light from an emitter into multiple raydirections is considered to be a form of optical crosstalk resulting inoverlapped light cones between adjacent viewing zones. In a number ofembodiments, emitters are jointly controlled to achieve desiredperformance from a light field display. In several embodiments, theemitters are jointly controlled to achieve a target light field. In anumber of embodiments, the emitters are jointly controlled to displayparticular views within specific viewing zones based upon informationincluding eye tracking information. In certain embodiments, the emittersare jointly controlled, subject to constraints related to minimum and/ormaximum permitted emitter intensities. Various joint control processesthat can be implemented in light field displays in accordance withvarious embodiments of the invention to reduce optical crosstalk arediscussed below.

In many embodiments, the emitters of an angular pixel are controlled sothat the intensity of the angular pixel in one or more specific raydirections is as close as possible to a target intensity. In severalembodiments, the matching of the emitted intensity with the targetintensity is achieved by controlling the intensities of each of thejointly controlled emitters so that the linear combination of theemitted intensities of the emitters in the one or more specific raydirections are as close as possible to their target intensities. As isdiscussed below, the one or more specific ray directions can bedetermined using eye tracking information to identify ray directionsthat contribute to displays in particular viewing zones that are visibleto viewers' eyes.

In a number of embodiments, a calibration process can be performed todetermine the contribution of individual emitters to specific raydirections. As is discussed further below, the calibration may alsoaddress the contribution to individual emitters to specific regions of aview displayed within one or more particular viewing zones. Thecalibration data can be used to define a mixture matrix for an angularpixel. In several embodiments, the mixture matrix maps emitterintensities to each ray direction. In many embodiments, the calibrationprocess used to develop mixture matrices is not performed with respectto each light field display. Instead, a representative light fielddisplay can be characterized to develop mixture matrices for its angularpixels. These mixture matrices can then be utilized directly by otherlight field displays or with modifications determined by a secondcalibration process specific to the particular light field displayutilizing the calibration information. In several embodiments, thecalibration data can be temperature dependent and temperature sensorsand/or other environment sensors are provided across the display toenable selection of appropriate calibration data.

When a mixing matrix is defined for an angular pixel, the intensities ofthe emitters that can achieve target intensities in a set of specificray directions can be determined by inverting the mixture matrix.Multiplying the inverted mixture matrix by a set of target intensitiesin specific ray directions can result in a set of emitter intensitiesthat are not physically achievable in the real world. For example,multiplying a set of target intensities in specific ray directions bythe inverted mixture matrix may result in negative intensities and/orintensities that are larger than the intensities that can actually begenerated by the emitters. In a number of embodiments, constraints areplaced on the permitted target intensities to increase the likelihoodand/or guarantee that the control process generates physicallyachievable emitter intensities. In certain embodiments, targetintensities are constrained to be between a minimum value that isgreater than zero and a maximum value that is less than the maximumachievable emitter intensity. The minimum target intensity alloweddetermines the black level of the display. Furthermore, the maximumtarget intensity allowed determines the dynamic range of the light fielddisplay.

In many embodiments, the emitter intensities of an angular pixel can bedetermined using an optimization process that seeks to minimize thedifference for a set of ray directions f between the target intensityL_(T)(ω) of that angular pixel in the ray direction w E f and theproduct of the emitter intensity L_(A)(x,ω) at particular emitterlocation x in ray direction ω multiplied by the mixture matrix A for theangular pixel. The optimization function can be formulated as follows:

$\min\limits_{\forall{\omega \in \Omega}}{\mathcal{L}(\omega)}$${where},{{\mathcal{L}(\omega)} = {\sum\limits_{x}\;{{Norm}\left( {{L_{T}(\omega)} - {A_{x,\omega}{L_{A}\left( {x,\omega} \right)}}} \right)}}}$

and where A_(x,ω) are the elements of the mixture matrix A.

The distance between a target intensity in a particular ray directionand the linear combination of the emitter intensities in that raydirection can be determined using any of a variety of distance metricsincluding (but not limited to) the L₁ norm and/or the L₂ norm. Incertain embodiments, the emitter intensities can be determined using aprocess that optimizes the emitter intensities based upon a neuralnetwork that is trained to generate an output indicative of perceivedimage quality. Any of a variety of objective functions can be utilizedin the optimization of emitter intensities of an angular pixel asappropriate to the requirements of specific applications in accordancewith various embodiments of the invention.

In several embodiments, the optimization is a constrained optimizationthat attempts to minimize the differences between the emittedintensities in specific ray directions and desired target intensities,but subject to specific minimum and maximum permitted targetintensities. In this way, image quality can be traded off against theblack level and/or the dynamic range of the display. The tradeoffbetween image quality and the black level and/or the dynamic range ofthe display can be determined dynamically in response to factorsincluding (but not limited to) the content displayed, the number ofviewers, and/or ambient lighting conditions. Furthermore, any of avariety of other alternative and/or additional constraints can beutilized to determine the manner in which to control emitter intensitiesof angular pixels in a light field display as appropriate to therequirements of specific applications in accordance with variousembodiments of the invention.

While much of the discussion above relates to controlling emitters in asingle angular pixel based upon desired target intensities in a set ofray directions, light field displays in accordance with many embodimentsof the invention can jointly control the emitters in multiple angularpixels based upon desired target intensities in a set of ray directions.A control process can be performed in which the sum of the contributionsin a particular ray direction w are determined are summed across theemitters of multiple angular pixels (as opposed to the emitters of asingle angular pixel). Typically, a number of angular pixels willcontribute to the perceived intensity of a pixel within a particularviewing zone. The number of angular pixels that contribute to theperceived intensity of a pixel within a particular viewing zone istypically dependent on the distance of the viewer from the light fielddisplay. For example, when the viewer is far from the light fielddisplay the number of angular pixels that include emitters thatcontribute to the perceived intensity of a pixel within a viewing zoneoccupied by an eye of the viewer is greater than when the viewer iscloser to the light field display.

In a number of embodiments, eye tracking information is utilized todetermine viewing zones in which tracked eyes are located (or are likelyto be located). This information can then be utilized to determine a setof ray directions for various angular pixels within the light fielddisplay that contribute to the views displayed in the viewing zones inwhich tracked eyes are or are likely to be located. As noted above, thedistance of the viewer from the light field display can cause the lightfield display to treat multiple angular pixels as contributing to theemitted intensity in a particular ray direction to reduce thecomputational complexity of jointly controlling the emitters within thelight field display. In addition, the target intensities for each raydirection in the set of ray directions for the various angular pixelscan be defined. The target intensities for the set of ray directions canthen be utilized to jointly control the emitters in one or more angularpixels to provide emitted intensities that achieve a pre-determinedobjective relative to the target intensities using a process similar toany of the processes outlined above.

While various processes are described above for jointly controllingemitters within a single angular pixel and/or set of angular pixels toachieve specific goals with respect to the display of one or more viewswithin one or more viewing zones, any of a variety of processes can beutilized to determine the manner in which to jointly control theintensities of emitters within one or more angular pixels as appropriateto the requirements of specific applications in accordance with variousembodiments of the invention.

Reducing Ghosting Using Eye Tracking

In light field imaging, a light field image is formed within a specificviewing zone by various angular pixels emitting light in a set of raydirections that combine to form a view within the viewing zone. Theangular pixel does not, however, emit light in a singular ray. Rather,the angular pixel emits a light cone in a specific ray direction. If thelight cones of neighboring ray directions do not overlap, it would beeasy to compute how much light each angular pixel should emit to bestapproximate the light field that the display should ideally create.However, in practice the light cones of neighboring ray directionsoverlap with increased distance from the light field display. As notedabove, some light may leak to other, non-obvious directions due toscattering and imperfections in the optical components of the display.

A form of optical crosstalk that is often referred to as ghosting orleakage can result when a viewer perceives multiple views displayed inoverlapping viewing zones. The term ghosting is typically utilized torefer to a phenomenon whereby a viewer's eye is located in overlappingviewing zones in which different views are displayed and so the viewerperceives a doubling of the image or “ghosting.” The term ghosting ismost often used in the context of stereoscopic displays to refer to aphenomenon where Left and Right images are presented to a viewer and atleast one of the viewer's eyes perceives both the Left and Right images(instead of the left eye seeing the Left image and the right eye seeingthe Right image in isolation). In the context of a multiviewautostereoscopic display, ghosting can manifest as a result of an eyeseeing multiple images from adjacent viewing zones. In a number ofembodiments, eye tracking information is utilized to reduce the extentto which viewers of a multiview autostereoscopic display experienceghosting.

The phenomenon of ghosting is conceptually illustrated in FIG. 11. Inthe illustrated example, emitters in an angular pixel 1100 produce lightcones 1100, 1102 that contribute to views displayed in adjacent viewingzones. In the illustrated example, the optical system of the angularpixel is assumed perfect and the two light cones 1101 and 1102 arenon-overlapping. In many embodiments, the light cones 1101 and 1102 arelikely to be overlapping. As the viewer's eye moves from the firstviewing zone 1104 to the second viewing zone 1106, the viewer's eyeoccupies a ghosting zone 1108 in which the eye simultaneously perceivesthe views displayed in the two adjacent viewing zones 1104, 1106. Theextent of the discomfort and/or perceived image degradation that resultsfrom the ghosting will typically depend upon the similarity between theviews displayed in the adjacent viewing zones. In a number ofembodiments, information regarding ghosting can be utilized to determinethe target intensities in specific ray directions and this informationcan then be utilized in processes similar to those described above tojointly control the emitters within the various angular pixels of thelight field display to reduce ghosting.

A process that can be utilized by a light field display to render theimages displayed in viewing zones based upon eye tracking information toreduce ghosting in accordance with an embodiment of the invention isconceptually illustrated in FIG. 12. The optical system of the angularpixel 1200 is shown as perfect and the two light cones 1201 and 1202 arenon-overlapping. As noted above, light cones 1201, 1202 are likely to beoverlapping in practically implementable light field displays. When theviewer's eye is located in a ghosting zone 1208, the eye simultaneouslyperceives the views displayed in the two adjacent viewing zones 1204,1206. In a number of embodiments, the light field display can ceasedisplaying different views in the two adjacent viewing zones 1204, 1206and instead display the same view in light cones 1210 and 1212 as theviewer's eye travels between viewing zones 1204 and 1206. As notedabove, the view that is displayed in the two adjacent viewing zones1204, 1206 can be rendered based upon the specific location of theviewer's eye to reduce aliasing. Alternatively, the light field displaycan render different views in the light cones 1210 and 1212 thatsuperimpose in such a way that the sum of the views creates a desiredview at the location of the viewer's eye.

FIG. 12 illustrates a simple example of ghosting reduction, inaccordance with an embodiment. In many instances, multiple viewing zonesoverlap and the intensities of the angular pixels of the light fielddisplay are determined so that the sum of the views perceived by aviewer's eye closely matches a desired view. Light field displaysdescribed herein can offer display modes including (but not limited to)a mode in which overlapping viewing zones display the same view renderedin a viewpoint dependent manner and/or a mode in which different viewsare displayed in overlapping viewing zones in a manner that results inthe sum of the views at a particular viewpoint corresponding to adesired view.

Referring now to FIG. 13, in a number of embodiments, uncertainty in eyetracking can be accommodated by using angular pixel 1300 to alwaysdisplay the same color and intensity 1301, 1302 in viewing zones 1304,1306 adjacent to tracked eyes in the manner illustrated in FIG. 13.Maintaining consistent color across adjacent viewing zones 1304, 1306can be achieved by controlling emitters within angular pixels so thatadjacent emitters display the same color, as an emitter used to displayan image within a viewing zone occupied by a tracked eye. In this way,even when the eye is located in viewing zone 1308 between adjacentviewing zones 1304 and 1306, the tracked eye sees consistent colors,thus reducing aliasing and other undesirable effects. While theconceptual illustration in FIG. 13 shows the color of the emitters asbeing invariant based upon the position of the viewer's eye, in manyembodiments the color varies as the viewer's eye moves to reducealiasing in the manner described above. The light field display (formedof a plurality of angular pixels 1300, simply controls adjacent emittersso that they produce the same color as the view that is displayedchanges based upon the tracked eye location.

Accordingly, light field displays in accordance with several embodimentsof the invention utilize a process similar to the process shown in FIG.10 above to render views for display in viewing zones occupied bytracked eyes and control the emitters so that the same view is displayedin adjacent viewing zones. While the embodiments discussed above withreference to FIGS. 12 and 13 illustrate two adjacent viewing zones, itshould be appreciated that light field displays in accordance with manyembodiments of the invention can control emitters so that viewsdisplayed in more than two adjacent or overlapping viewing zones are thesame for reasons including (but not limited to) the reduction and/orelimination of ghosting.

While much of the discussion above focuses on the reduction and/orelimination of ghosting, another advantage of displaying the same viewin adjacent viewing zones/controlling adjacent emitters to produce thesame color is that the light field display's image quality becomes morerobust to errors in eye tracking information and/or intra-microlenscrosstalk. When the same view is displayed in adjacent viewing zones,errors in the eye tracking information do not carry an inherent risk ofghosting (e.g., due to the light field display being unaware that theviewer's eye is actually located within a ghosting zone and perceivingtwo different views). In addition, intra-microlens crosstalk betweenadjacent emitters (discussed in further detail below) does not result ina degradation of the perceived image quality as the emitters produce thesame colors.

While a number of processes are described above for addressing ghosting,any of a variety of techniques for addressing ghosting can be utilizedas appropriate to the requirements of specific applications inaccordance with various embodiments of the invention. In a number ofembodiments, calibration information concerning the extent of theoverlap between adjacent viewing zones can be utilized to render viewsin adjacent viewing zones that combine in a manner that creates aconsistent view for the viewer. In this way, the superposition of theoverlapping views formed by adjacent emitters can be taken into accountin the rendering of views based upon the location of the tracked eye sothat the intensity of particular angular pixels is uniform as the eyemoves across the field of view of the viewing zones.

Foveated Light Field Rendering Using Gaze Tracking

Computational efficiencies can be achieved when light field displaysincorporate gaze tracking hardware in accordance with variousembodiments of the invention. In certain embodiments, gaze trackinginformation can be used to further define the set of ray directionsutilized in the joint control of emitters within a light field displayto support rendering of foveated views based upon gaze direction as isdiscussed further below. As noted above in reference to FIG. 6A, eyetracking involves tracking the locations of a viewer's eyes. As shown inFIG. 6B, gaze tracking involves tracking not only where a viewer's eyesare located, but where the viewer is looking. The computationalcomplexity of rendering views for display in viewing zones likely to beperceived by the viewer's eyes (and potentially adjacent viewing zones)can be reduced by only rendering the views with full resolution anddisparity information in a foveated region corresponding to the regionof the display where the viewer's gaze is likely directed. That is,while the tracking of the viewer gaze may be more involved than eyetracking, the incorporation of gaze tracking into a light field displaycan reduce the computational complexity of the image rendering byrestricting the light field views rendered to only those locations atwhich the viewer is gazing.

As known in studies, human sensitivity to color information is highestin the center of the retina, generally referred to as the fovea, whileretinal acuity and color sensitivity is typically poor away from thecenter of the retina. However, the ability to sense rapid changes inintensity or movement is highly developed outside the center of theretina. Human sensitivity to stereopsis is highest in a central cone ofvision (typically 10°) around the axis of vision; outside this area,depth cues can be reduced for high-frequency content without asignificant reduction in the perceived quality of the display.Accordingly, light field displays can use gaze tracking information toidentify foveated regions of the views to be rendered for display in theviewing zones occupied by a viewer's eyes. Within the foveated region,i.e., those views viewed by the center of the retina, the views can berendered at the maximum supported resolution and with stereopsis.Outside the foveated region, the resolution of the image can bedecreased and stereopsis need only be maintained with respect to largefeatures or low-frequency content. Gaze information can also be utilizedfor tone mapping and/or application of depth dependent blur based uponthe specific point within a scene to which a viewer's gaze is directed.

In many embodiments, a light field display that employs foveatedrendering is able to change the rendered resolution and/or disparity ofdifferent regions of the rendered views during saccadic suppression(e.g., when a tracked eye blinks the light field display can acquire newgaze tracking information and render new foveated views prior to thecompletion of the viewer's saccadic suppression). Alternatively, thelight field display can gradually adjust the resolution and/or extent ofstereopsis in different regions so as to avoid notice.

In a number of embodiments, light field displays employ foveatedrendering in a manner that accommodates uncertainty in the location oftracked eyes and in the determination of the gaze direction of thetracked eyes. Retinal acuity, color sensitivity, grayscale resolutionand sensitivity, and time resolution and motion sensitivity with retinalposition can be represented as a feature vector. When informationconcerning the 3D eye position and 2D gaze direction are known, theretinal feature vector can be mapped to a vector field in a 4D rayspace. Each ray terminating on the retina and passing through the pupilcan be assigned a feature vector value related to the retinal point atwhich the ray terminates. All rays that do not terminate on the retinacan be assigned a vector value of zero. By assigning values in a mannerthat is dependent upon characteristics of the retina including (but notlimited to) acuity and/or color sensitivity, the light field display canrender views with the most detail and color on rays originating fromangular pixels where the eyes are looking and passing through the pupil.Using the same process, less detail and color can be rendered in regionsof views that corresponding to rays originating from angular pixels thatare not at the center of a viewer's gaze or that are unlikely to passthrough a viewer's pupil. In several embodiments, the renderingaccommodates the eye's sensitivity to sudden changes in intensity withinperipheral vision by providing smooth transitions for rays outside thecenter of the viewer's gaze. In certain embodiments, the distance of theviewer from the light field display is also considered and views arerendered with a level of detail appropriate to the level of detail thatcan be observed at the distance the viewer is located.

If gaze direction and eye position are not exactly known, the gazedirection and eye position can be represented as a 5D probabilitydistribution (i.e., 3-dimensions related to position and 2-dimensionsrelated to gaze) as illustrated in FIG. 14. In many embodiments foveatedrendering can take uncertainty into account by weighting the featurevector field for each eye position and gaze direction with the 5Dprobability density function and then integrating over the 3D eyepositions and 2D gaze directions. The result is a feature vector fieldφ(x,y,u,v) in a 4D ray space defining the importance of each ray fromthe perspective of all the features. Views can then be rendered byrendering rays based on this resulting feature vector functionφ(x,y,u,v). The feature vector fields can be aggregated for the multipleeyes of one or more viewers using an aggregation function such as (butnot limited to) maximum, mean, or minimum.

A process that can be utilized to perform foveated rendering based uponeye and gaze tracking information in a manner that accounts foruncertainty in eye and gaze tracking information in accordance with anembodiment of the invention is conceptually illustrated in FIG. 15. Theprocess 1500 accepts as inputs eye positions and gaze directionsobtained from eye and gaze tracking systems 1502, 1503, respectively. Ina number of embodiments, confidence information with respect to the eyeand gaze tracking information can be utilized to generate eye and gazeprobability distributions using eye position and gaze directionprobability models 1504, 1505, respectively. These distributions can beutilized to generate a combined eye position and gaze directionprobability model 1506 that can be utilized in the generation of controlinformation for angular pixels within the light field display. Theprocess of generating the control information involves, for example,using the combined eye position and gaze direction probabilityinformation from the eye and gaze probability model 1506 to determinethe importance of rays within the light field using a light fieldimportance calculator 1508 to generate a feature vector functionI(x,y,u,v). A light field processor 1510 can combine the feature vectorfunction I(x,y,u,v) with scene content 1512 provide view requests toview renderers 1514, which then renders and returns views to light fieldprocessor 1510. These views are then utilized by light field processor1510 to generate control information as specific pixel values for theangular pixels of the light field display.

While specific processes are described above with reference to FIG. 15for rendering foveated views based upon gaze tracking information, anyof a variety of processes can be utilized to render views having varyingresolution and representation of depth cues to increase thecomputational efficiency of the light field display as appropriate tothe requirements of specific applications in accordance with variousembodiments of the invention. In systems in which only head or eyetracking information is available (and not gaze tracking), the lightfield display can make decisions based upon the complexity of the viewsto render based upon the ability of the viewer to perceive image detailand depth queues. In a number of embodiments, the resolution andrepresentation of depth cues varies based upon the distance of theviewer from the display. In other embodiments, any of a variety offactors can be considered in determining the level of detail and/ornumber of views that are rendered for display by a light field display.

While various processes for sampling scenes and rendering images fordisplay based upon the tracked locations of the eyes of one or moreviewer are described above with reference to FIGS. 10-15, any of avariety of processes can be utilized to modify the view displayed withina viewing zone of a light field display or conventional multiviewautostereoscopic display based upon the location of a tracked eye withinthe viewing zone as appropriate to the requirements of a givenapplication in accordance with various embodiments of the invention. Forexample, light field displays in accordance with several embodiments ofthe invention utilize a combination of anti-aliasing pre-filtering andeye-location-dependent rendering. In several embodiments, the lightfield display does not immediately fall back to conventional multiviewautostereoscopic or 2D display in response to a loss of eye tracking butcan estimate eye location based upon previously tracked eye locations.In certain embodiments, eye location predictions are also utilized torender views that will be displayed at a future time using predicted eyelocations. Furthermore, the benefits of tracking eyes are not limited toincreasing the perceived depth of field of a light field display. Eyetracking information can also be utilized to reduce the energy,processing, and/or bandwidth requirements of light field displays as isdiscussed further below.

Reducing Energy, Processing and/or Bandwidth Using Eye TrackingInformation

Light field displays in accordance with many embodiments of theinvention can use eye tracking information to reduce energy consumptionand/or reduce the amount of bandwidth required between the processingsystem and the light field display's backplane to drive the display. Inaddition to eye tracking information providing information regardingviewing zones of the light field display that contain tracked eyes, theeye tracking information can identify viewing zones that do not containtracked eyes. As noted above, many eye tracking systems provide both eyetracking information and confidence metrics that indicate the likelihoodthat an eye is located in a particular location. Accordingly, the lightfield display can determine that an eye is absent within a viewing zoneand make decisions accordingly. In many embodiments, the light fielddisplay can save energy by turning off emitters that would otherwise beused to display views within the unoccupied viewing zones. Anotherbenefit of deactivating emitters is an improvement in image quality dueto a reduction in crosstalk. Significant processing efficiencies canalso be achieved by only rendering views for viewing zones containingtracked eyes. In addition, the bandwidth required between the lightfield display's processing system and the backplane of the light fielddisplay can be reduced by only transmitting control information foractive emitters (e.g., emitters that are utilized to display a viewwithin a viewing zone occupied by a viewer's eye). Given fixed totalbandwidth between a processing system and the backplane of a light fielddisplay, the bandwidth may be allocated based upon a set of viewpointsthat have been identified with confidence in a manner that controls theangular and spatial resolution of views rendered within viewing zonesacross the field of view of the light field display to provide highperceived quality to the tracked viewers.

Energy-Efficient Control of Emitter Activation Using Eye TrackingInformation

A process for controlling the activation of emitters within a lightfield display in accordance with an embodiment of the invention isconceptually illustrated in FIG. 16. In the illustrated embodiment 1600,two sets of viewers 1602 and 1604 are shown and a display plane 1610 ofa light field display generates multiview autostereoscopic views 1612,1614, 1616, and 1618 within the viewing zones occupied by each of theviewers' eyes. The light field display exploits information concerningviewing zones that are not occupied by viewer eyes (indicated by Xmarkings 1620) to switch off the emitters responsible for displayingviews within those viewing zones.

In a number of embodiments, the light field display also displays viewsin viewing zones adjacent viewing zones occupied by tracked eyes asconceptually illustrated in FIG. 17. As shown in FIG. 17, embodiment1700 includes a viewer 1702 viewing multiview autostereoscopic views1712 and 1714 generated by a display plane 1710 of a light field displaywhile the emitters contributing to views outside of the viewing zones(indicated by X markings 1720) are switched off or the views therein arereduced in resolution and/or brightness. In this case, display plane1710 also generates views in a zone 1730 around the likely eye positionsof viewer 1702. Some uncertainty may exist with respect to the locationof tracked eyes, and light field displays in accordance with manyembodiments of the invention will also render views for the viewingzones that are adjacent to the viewing zones occupied by tracked eyes,as in the example illustrated in FIG. 17. That is, even if there areuncertainties in the specific location of the viewer's eyes, a systemsuch as shown in FIG. 17 can accommodate the uncertainty while stillproviding savings in energy and computation.

In addition, light field displays in accordance with several embodimentsof the invention respond to a loss of eye tracking by rendering views inviewing zones that are likely to be occupied by previously tracked eyesbased upon the tracked trajectory of the eyes prior to the loss oftracking. In many embodiments, the light field display displays lowerresolution autostereoscopic views and/or 2D content (e.g., a single viewthat does not vary between viewing zones) in many or all of the viewingzones that are not occupied by tracked eyes as conceptually illustratedin FIG. 18. For example, in embodiment 1800 in FIG. 18, a viewer 1802 isviewing multiview autostereoscopic views 1812 and 1814 generated by adisplay plane 1810 of a light field display. Outside of views 1812 and1814, a uniform 2D content or a lower resolution light field images areprovided by display plane 1810 in a region 1840 outside of the viewer'seyes. In addition, the views in region 1840 can be refreshed at a lowerframe rate to reduce the computational requirements of rendering viewsin viewing zones in which they are unlikely to be observed by a viewer.It is recognized herein that the specific manner in which a light fielddisplay responds to eye tracking uncertainty and/or an absence of atracked eye within a particular viewing zone is largely dependent uponthe requirements of a given application.

In certain embodiments, the light field display is configured to track aspecific viewer's eyes and preserves the privacy of the display by onlygenerating multiview autostereoscopic views within the viewing zonesoccupied by the viewer's eyes and potentially in adjacent viewing zones.In several embodiments, the presence of the eyes of an unauthorizedviewer can be detected and the light field display can actively preventthe display of views in viewing zones occupied by the eyes of anunauthorized viewer. In many embodiments, the location and image of aviewer's head is utilized to determine whether a viewer is an authorizedor an unauthorized viewer. In a number of embodiments, facialrecognition and/or a fiducial can be utilized to differentiate betweenauthorized and unauthorized viewers.

Reducing Pixel-Level Optical Crosstalk Using Eye Tracking Information

Reducing the number of active emitters in a light field display basedupon eye tracking information can further improve image quality byreducing both intra-microlens and inter-microlens crosstalk. Forinstance, internal reflections and scattering within the optical systemof an angular pixel can cause crosstalk both within the angular pixel(which can be referred to as intra-microlens crosstalk) and betweenangular pixels (which can be referred to as inter-microlens crosstalk).

An example of a manner in which intra-microlens crosstalk, i.e.,crosstalk between light emission from light emitters within a singleangular pixel, can manifest is conceptually illustrated in FIG. 19A. Anexample system 1900 includes a plurality of angular pixels 1902, asshown in FIG. 19A. A first emitter 1904 within one of angular pixels1902 emits a cone of light, represented by dashed arrows 1906. While themajority of the rays within the cone of light are directed towardspecific viewing zones by optical system 1908 (e.g., microlens array) ofthe angular pixel, stray rays 1910 can be reflected or refracted withinand at interfaces between adjacent angular pixels and coupled into alight cone (represented by solid arrows 1920) emitted by a secondemitter 1922 within system 1900. In this way, a viewer with an eye in aviewing zone that intersects the light cone 1920 of the second emitter1922 will perceive a mixture of the light from the first emitter 1904and the second emitter 1922. It is recognized herein that the ability toswitch off the first emitter 1904 when there are no tracked eyes inviewing zones that receive light from the first emitter enables thelight field display including angular pixels such as shown in FIG. 12 toreduce intra-microlens crosstalk within viewing zones that receive lightfrom the second emitter 1922.

The manner in which inter-microlens crosstalk, i.e., crosstalk betweenlight emission from different angular pixels, can manifest isconceptually illustrated in FIG. 19B. An example system 1950 includes aplurality of angular pixels 1952, as shown in FIG. 19B. A first emitter1954 in one of angular pixel 1952 produces a light cone, represented bysolid lines 1956, that is directed toward specific viewing zones byoptical system 1958 (e.g., microlens array) of the angular pixel. Aportion of the light cone 1956 is reflected at an air-microlensinterface as reflected light (represented by lines 1960). The reflectedrays can also be reflected within the light field display and emergefrom a neighboring angular pixel, as represented by thin arrows 1962.Further internal reflections may result in additional stray light fromneighboring angular pixels, such as represented by dashed arrows 1964.That is, the ability to switch off the first emitter 1954 when there areno tracked eyes in viewing zones that receive light from the firstemitter enables the light field display to reduce inter-microlenscrosstalk in adjacent angular pixels.

The extent to which both intra-microlens and inter-microlens crosstalkare generated by a light field display is largely dependent upon theamount of light that is reflected or scattered within the angular pixelswithin the light field display. Accordingly, only turning on emittersthat contribute to views within viewing zones occupied by tracked eyescan significantly reduce the amount of intra-microlens andinter-microlens crosstalk. With specific regard to intra-microlenscrosstalk, reducing the number of active emitters within each angularpixel can significantly decrease the amount of light that is internallyreflected within the optical systems of the angular pixels. Accordingly,the ratio of reflected or scattered light to directly emitted lightperceived at a viewer's eyes is reduced compared to a system in whichall emitters are active at all times. Similarly, reducing the number ofactive emitters within an angular pixel reduces the amount of light thatis reflected or scattered into adjacent angular pixels. Therefore, theratio of reflected or scattered light to directly emitted lightperceived at a viewer's eyes is similarly reduced compared to a systemin which all emitters are active. Furthermore, as discussed above,turning off emitters that are not contributing to views presented at thetracked eye locations can also lead to energy and computationalbandwidth savings, thus improving the efficiency of the light fielddisplay.

Controlling Emitter Activation Using Eye Tracking Information

Specifically addressing the computational bandwidth required ingenerating light field views, the computational load on the processingsystem as well as the internal bandwidth requirements of a light fielddisplay increases with the number of views that are rendered and theframe rate of the display. Furthermore, the energy requirements of thelight field display increase with the number of viewing zones.Accordingly, the processing, bandwidth and/or energy requirements oflight field displays in accordance with many embodiments of theinvention can be significantly impacted by decisions concerning whetherto render views for display only in specific viewing zones correspondingto tracked eyes, the number of viewing zones for which views arerendered, whether to render and display 2D or 3D views in additionalviewing zones, the number of viewing zones in which to display the 2Dview, and/or the number of viewing zones that are deemed inactive. Thespecific manner in which a light field display in accordance withvarious embodiments of the invention is configured to render views fordisplay in active viewing zones based upon eye tracking information islargely dependent upon the capabilities of the display and therequirements of a given application.

A process for controlling activation of emitters based upon eye trackinginformation in order to display views in viewing zones occupied bytracked eyes in accordance with an embodiment of the invention isillustrated in FIG. 20. A process 2000 begins with a start step 2001,followed by a step 2002 to track the eyes of a viewer using, forexample, an eye tracker arrangement integrated with the light fielddisplay, such as shown in FIG. 2. Then, a determination 2004 is madethat the eyes of a viewer are located. The eye tracking information canbe utilized to identify the viewing zones occupied by tracked eyes andthe light field display processing system can render 2006 views fordisplay in the identified, active viewing zones. For instance, the viewscan be viewpoint dependent and rendered specifically for the viewingzone to preserve stereo and motion parallax for the viewer. Optionally,if calibration data, such as factory settings for the emitters of thearray or angular pixels in the display area or other calibrationinformation as described above, are stored in a memory of the display,such as on a processing unit, the calibration in formation can beincorporated in a step 2007 into the rendered views from step 2006. In astep 2008, the rendered views are then used to generate controlinformation for the emitters in the angular pixels to actually generatethe views by the display. A determination 2010 is made to determinewhether the eye tracking and view rendering process is complete. If theanswer to determination 2010 is YES, the process is completed, thenprocess 2000 proceeds to an end step 2012. If determination 2010concludes that the eye tracking and view rendering process is not yetcomplete, the process returns to step 2002 to attempt to track the eyesof the viewer.

When the light field display loses eye tracking such that determination2004 resulting in the answer that the eyes have not been located, theprocessing system of the light field display can render 2020 views forviewing zones that are likely to be occupied by a viewer's eyes basedupon prior tracking information or other a priori information. Forinstance, the processing system can render or utilize a separatelyrendered 2D view or a lower resolution view and display the view inviewing zones likely to be occupied by a viewer's eyes based upon priortracking information. The relevant control information is then generatedin step 2008, and the process again proceeds to determination 2010 todetermine whether the process is completed.

In a number of embodiments, views can also be rendered for viewing zonesadjacent to the viewing zones occupied by tracked eyes, such asillustrated in FIG. 17. In several embodiments, these views outside ofthe viewing zones occupied by tracked eyes can be rendered at lowerresolution and/or with less detailed depth cues to reduce thecomputational overhead associated with rendering additional views inviewing zones that may not be occupied by viewers' eyes. In certainembodiments, a 2D view can also be rendered for display in some or allof the viewing zones that are not occupied by tracked eyes, asillustrated in FIG. 18. In many embodiments, the light field displaydoes not display views in a number of the viewing zones that areunoccupied by tracked eyes, as illustrated in FIG. 16. Additionally, thelight field display may have different modes and/or employ decisionlogic to determine whether to activate emitters to display a 2D view orto switch off emitters when a viewing zone is unoccupied by a trackedeye, as described above, as appropriate to the requirements of a givenapplication.

The rendered views can then be utilized to generate 2008 controlinformation to activate emitters within the light field display tocontribute to the appropriate light field views. In many embodiments,the active emitters are the emitters utilized to display views. As notedabove, a viewing zone can display a view specifically rendered for theviewing zone (e.g., when a particular viewing zone contains a trackedeye) or a view that is displayed in multiple viewing zones (such as a 2Dview or a lower resolution view). As the bandwidth required for theprocessing system to provide the control information to the backplane(i.e., control circuitry) of the light field display is typicallydependent upon the number of views displayed by the light field display,the bandwidth is reduced when control information is provided only foractive emitters contributing to views in the active viewing zones. Inseveral embodiments, the processing system provides a view that iscommon to multiple viewing zones (2D view) with metadata indicating theviewing zones in which the 2D view is displayed. In many embodiments,the amount of information that is transmitted to the backplane can bereduced by enabling the light field display controller to control asmaller number of logical emitters than there are physical emitters in agiven angular pixel. In this way, the bandwidth requirements between thecontroller and the backplane of the emitter array of the light fielddisplay can be reduced. Furthermore, circuitry within the backplane canbe made responsible for interpolating control information for logicalemitters to control information for a larger number of emitters withinan angular pixel.

By way of example, there can be 500×500 physical light emitters that incombination with a lenslet form an angular pixel. The controller,however, may only be able to control emitters corresponding to 100directions. For example, a unique color could be specified for eachlogical emitter in a 10×10 grid of logical emitters corresponding to the500×500 physical emitters, and the corresponding control information isthen utilized by the backplane to interpolate values within a 50×50subgrid of physical emitters corresponding to each logical emitter. Inanother control mode, the controller can address a specific set oflogical emitters corresponding to the 100 logical emitters correspondingto the physical emitters contributing to views perceived by the trackedeyes of a viewer. The backplane control circuitry can then take thecontrol information corresponding to these 100 logical emitters andcontrol the activation and/or interpolate the color of the remainingphysical emitters in the angular pixel. The specific logical emittersthat are addressed by the controller can be addressed in a number ofways including (but not limited to) providing an index and/or providinga subwindow within the larger array of logical emitters controlling thephysical emitters forming the angular pixel. Furthermore, controlinformation between the controller and the backplane can be run lengthencoded to further reduce the bandwidth requirements of the controlinformation passing between the controller and the backplane.

The numbers of logical and physical emitters referenced above areprovided merely by way of example, and light field displays can useangular pixels that include arrays of any number of emitters. Inaddition, the specific control information describing the activeemitters and associated metadata provided by the processing system tothe backplane of the light field display is largely dependent upon therequirements of a specific application. Further, the specific manner inwhich the light field display can respond to losing tracking of one orboth of a viewer's eyes typically depends upon the requirements of agiven application. Also, the availability of gaze tracking informationin addition to eye tracking information can enable additional efficiencygains as is discussed further below.

While a number of processes for rendering views for display via a lightfield display and for providing control information for active pixels tothe control circuitry of a light field display based upon eye trackingand/or gaze tracking information are described above with reference tothe figures, any of a variety of techniques can be utilized forrendering views and determining viewing zones in which to render viewsbased upon head, eye, and/or gaze tracking information as appropriate tothe requirements of a given application in accordance with variousembodiments of the invention.

Accelerated View Synthesis Using Post-Rendering Image Warps

In many instances, the time taken to directly render a view of a scenefrom a 3D model of the scene for display in each viewing zone occupiedby a tracked eye may be too slow to accommodate a desired display framerate. In addition, eyes can move much faster than the display refreshrate. Therefore, eye position can change from the point in time at whicha light field display commences rendering a view for the eye and thepoint in time at which the rendering is complete and the view isdisplayed to the eye. Accordingly, light field displays in accordancewith many embodiments of the invention employ an image renderingpipeline that utilizes a cascade of successively simpler and fasterrenderers running on a parallel processing unit such as (but not limitedto) a GPU to render views based upon eye tracking information at adesired frame rate in a manner that increases the likelihood that thedisplayed view is appropriate to the viewpoint of a viewer's eyes at thepoint in time at which the display is refreshed. In a number ofembodiments, an initial renderer is implemented on the parallelprocessing unit that continuously produces sets of 2D views withaccompanying depth maps. These 2D views with accompanying depth maps canthen be utilized to perform much faster post-processing 3D and/or 2Dwarps and translations so that the final view that is displayed in theviewing zone occupied by a viewer's eye is rendered from a viewpointthat corresponds as closely as possible to the location of the viewer'seye within the viewing zone at the time the view is displayed. In otherwords, at each point within the view rendering pipeline, the latest eyetracking information is fed into the processing performed at that point,thus providing faster and more accurate view rendering for the trackedeye location.

A view rendering pipeline 2100 in accordance with an embodiment of theinvention is conceptually illustrated in FIG. 21. The view renderingpipeline 2100 receives eye tracking information concerning the locationof the tracked eyes of a viewer from an eye tracker process 2102. Theeye tracking information can refresh at an extremely high rate (e.g., 30kHz) using currently available equipment. An initial 3D renderingprocess 2104 can be performed on a parallel processing unit in whichsets of 2D images with accompanying depth maps are generated. In manyembodiments, the renderer generates a set of images at a rate that issimilar to the frame rate of the display (e.g., 24 Hz, 30 Hz, 60 Hz,etc.). In several embodiments, the set of images and accompanying depthmaps includes a number of images and accompanying depth maps that isgreater than the number of detected viewers. The rendering process cangenerate an image and depth map from the viewpoint of a detected eye (ora nearby viewpoint) and additional viewpoints. As discussed below, theadditional viewpoints enable interpolation and warping to produce afinal view that is displayed within a viewing zone occupied by thetracked eye. In several embodiments, the rendered 2D images correspondto views that are larger, and have higher resolution, than the viewsthat can be displayed by the light field display in order to accommodatesubsequent warp and translation processes.

Continuing to refer to FIG. 21, prior to display, a series ofpost-rendering warps and translations are performed using the set of 2Dimages and accompanying depth maps from 3D render process 2104 basedupon updated eye tracking information to generate the final views thatare displayed. In the illustrated embodiment, a post-rendering 3D warpprocess 2106 is performed that fuses adjacent 2D images based upon theiraccompanying depth information to form updated 2D images andaccompanying depth maps from viewpoints corresponding to the updatedlocations of the tracked eyes. In many embodiments, the post-renderingwarps and translations can be performed using a process that is an orderof magnitude faster than the initial rendering process.

As additional eye tracking updates are received by the processingsystem, the view rendering pipeline 2100 can continue to adjust therendered views prior to display using even faster processes. In thisillustrated embodiment, a 2D warp process 2108 is utilized to apply 2Dwarps and translations to adjust the updated 2D images in response tofurther updated eye tracking information. In several embodiments, the 2Dwarp process 2108 is an order of magnitude faster again than the 3D warpprocess 2106. These 2D warped images can then be finally adjusted usingan extremely fast 2D translation process 2110 based upon final eyetracking locations immediately prior to being sent as an output to bedisplayed from the light field display. In certain embodiments, the 2Dtranslation process 2110 can be an order of magnitude faster again thanthe 2D warp process 2108 and capable of performing the 2D warp processat a rate comparable to the sampling rate of the eye tracking process.In this way, the views that are displayed in specific viewing zones arerendered from viewpoints that correspond as closely as possible to themost recently tracked locations of each viewer's eyes. As noted above,the ability to render views that are specific to the location of theviewer's eyes as opposed to simply the specific viewing zone occupied bythe viewer's eyes can significantly decrease the extent of thedepth-dependent aliasing and other artifacts perceived by the viewer.

While specific processes are described above for rendering views usingparallel processing units in a manner that is dependent upon updated eyetracking information with reference to FIG. 21, any of a variety ofrendering processes and/or pipelines can be utilized as appropriate tothe requirements of specific applications and/or the manner in which thevideo content is encoded. In a number of embodiments, the video contentis generated using 3D models of a scene. In several embodiments, thevideo content is encoded using a multiview video codec that providesmultiple encoded 2D images and one or more uncompressed or compresseddepth maps and/or occlusion maps. Accordingly, the manner in which animage processing pipeline renders an initial set of 2D images andaccompanying depth maps that can be utilized in subsequent 3D and/or 2Dwarp and translation processes is largely dependent upon therequirements of a specific application.

While the above description contains many specific embodiments of theinvention, these should not be construed as limitations on the scope ofthe invention, but rather as examples of particular embodiments thereof.For example, the above discussion frequently references light fielddisplays that incorporate eye trackers. As can readily be appreciated,many of the systems and processes described above can be implemented inconventional multiview autostereoscopic displays and/or within displaysthat capture head tracking and/or gaze tracking information.Additionally, if facial recognition or other ways to distinguish betweendifferent viewers are integrated into the eye tracking system, thenexplicit user recognition and rendering of user-specific, personalizedcontent (i.e., not viewable by other users) can be implemented. Anotherextension of the above described concepts, as a subset of a fullautostereoscopic viewing experience, would be to show tracked stereoviews (i.e., only two views) on the multiview display for single ormultiple users.

To reiterate, some of the aspects of the embodiments described aboveinclude, for example:

-   -   The combination of a multiview autostereoscopic display        integrated with an eye tracking system including one or more eye        trackers, providing view rendering based on the tracked position        of at least one viewer's eyes.    -   The multiview autostereoscopic display integrated with an eye        tracking system, where the display includes an array of angular        pixels, each angular pixel including a plurality of emitters. In        some embodiments, the angular pixel includes one or more        microlenses and other optical elements.    -   The multiview autostereoscopic display integrated with an eye        tracking system, where the eye tracking system tracks the eyes        of at least one viewer using a hierarchical tracking process.    -   The multiview autostereoscopic display integrated with an eye        tracking system, where the image resolution, density of views,        and the type of views (two- or three-dimensional) of the        rendered view depends on the tracked eye positions.    -   The multiview autostereoscopic display integrated with an eye        tracking system, where views outside those viewing zones in        which eyes have been located are turned off.    -   The multiview autostereoscopic display integrated with an eye        tracking system, where views with higher resolution or density        of views are presented at at least one foveated region in which        an eye or gaze has been tracked.    -   The multiview autostereoscopic display integrated with an eye        tracking system, where the eyes of multiple viewers of the        display are tracked. In some embodiments, one eye tracker tracks        the eyes of multiple viewers. In other embodiments, multiple eye        trackers are used to track the eyes of multiple viewers.    -   The multiview autostereoscopic display integrated with an eye        tracking system, where different views are presented in        different viewing zones, according to the eye tracking        information. Higher resolution or higher density of views can be        presented in the viewing zones in which eyes of viewer(s) have        been located. Lower resolution or lower density or views,        two-dimensional only views, or even no view can be presented in        viewing zones in which eyes of viewer(s) have not been located.        The decision to modify the views presented in specific viewing        zones can also take into account the confidence with which the        eye tracking information is considered accurate. In this way,        savings in the display energy consumption and/or image        processing and data transfer bandwidth can be achieved.    -   The multiview autostereoscopic display integrated with an eye        tracking system, where the processing system is further        configured for optimizing the control information sent to        control the angular pixels for mitigating at least one of a        variety of factors affecting the image quality, such as optical        crosstalk between viewing zones (e.g., ghosting, aliasing) and        pixel-level optical crosstalk that may result from non-ideal        optical elements and/or emitter performance.    -   The multiview autostereoscopic display integrated with an eye        tracking system, where a view rendering pipeline within the        processing system takes into account updated eye tracking        information during the view rendering process so as to reduce        the computational bandwidth required in the view rendering        process while improving the image quality as seen by the viewer.

Accordingly, the scope of the invention should be determined not by theembodiments illustrated, but by the appended claims and theirequivalents.

What is claimed is:
 1. A multiview autostereoscopic display forproviding a plurality of views to a viewer, comprising: a display areacomprising an array of angular pixels, where: each angular pixel isconfigured for emitting light that varies across a field of view of thatangular pixel; and the array of angular pixels is configured fordisplaying different views in different viewing zones across a field ofview of the display; at least one eye tracker configured for detectingthe presence of eyes of the viewer within specific viewing zones andproducing eye tracking information related to locations of the eyes sodetected within the specific viewing zones; and a processing system,where the processing system is configured for: rendering a specific viewfor each one of the eyes so detected based upon the location of that eyewithin the viewing zone in which that eye was detected; and generatingcontrol information for the array of angular pixels to cause thespecific view for that eye to be displayed in the viewing zone in whichthat eye was detected.
 2. The multiview autostereoscopic display ofclaim 1, wherein the processing system is configured for rendering anupdated view for each one of the eyes so detected based upon an updatedlocation for that eye within the viewing zone in which that eye waspreviously detected.
 3. The multiview autostereoscopic display of claim2, wherein the processing system is configured for rendering the updatedview based upon a relative motion between a previous location for thedetected eye and the updated location for the detected eye.
 4. Themultiview autostereoscopic display of claim 1, wherein the at least oneeye tracker is configured for detecting locations of the detected eyeswithin the specific viewing zones by performing a hierarchical eyetracking process comprising determining a location of the viewer,determining a location of a head of the viewer based on the location ofthe viewer so determined, and determining locations of the viewer's eyesbased upon the location of the head of the viewer so determined.
 5. Themultiview autostereoscopic display of claim 1, wherein the processingsystem is configured for rendering at least one of reduced resolutionviews, reduced density views, and two-dimensional views for display inspecific viewing zones based upon eye tracking information.
 6. Themultiview autostereoscopic display of claim 5, wherein the eye trackinginformation includes confidence information indicating a level ofuncertainty associated with a tracked eye location, and wherein theprocessing system is further configured for adjusting the renderingaccording to the confidence information.
 7. The multiviewautostereoscopic display of claim 1, wherein the processing system isconfigured for generating control information to control activation ofemitters within the array of angular pixels based upon eye trackinginformation to prevent display of views in specific viewing zones. 8.The multiview autostereoscopic display of claim 1, wherein theprocessing system is configured for rendering specific views for eachone of the eyes so detected based upon the location of that eye withinthe viewing zone in which that eye was detected and in mannerconstrained by a determined viewer specific inter-pupil distance.
 9. Themultiview autostereoscopic display of claim 1, wherein: the at least oneeye tracker comprises at least one gaze tracker configured fordetermining a gaze direction for each detected eye of the viewer; andthe processing system is configured for rendering a specific view foreach detected eye based upon the location of the detected eye within theviewing zone in which the eye was detected and the gaze direction of thedetected eye.
 10. The multiview autostereoscopic display of claim 1,wherein: the processing system is configured for generating controlinformation for a plurality of logical emitters; and the display areafurther comprises backplane circuitry configured for receiving thecontrol information for the plurality of logical emitters, andinterpolating the control information for the plurality of logicalemitters to control information for the array of angular pixels in thedisplay area.
 11. The multiview autostereoscopic display of claim 1,wherein the multiview autostereoscopic display is configured forproviding views viewable by a first viewer and a second viewer, whereinthe at least one eye tracker is further configured for simultaneouslytracking eyes of the first and second viewers, and wherein theprocessing system is further configured for: rendering specific viewsbased upon a first scene for each detected eye of the first viewer,rendering additional views based upon a second scene for each detectedeye of the second viewer, the second scene being at least partiallydifferent from the first scene, and generating control information forthe array of angular pixels such that, simultaneously, the specificviews for each detected eye of the first viewer are displayed in viewingzones in which the eyes of the first viewer were detected, and theadditional views for each detected eye of the second viewer aredisplayed in viewing zones in which the second viewer's eyes weredetected.
 12. The multiview autostereoscopic display of claim 11,wherein the control information so generated by the processing systemcauses no view to be displayed outside the viewing zones in which atleast one of the first and second viewers' eyes were detected.
 13. Themultiview autostereoscopic display of claim 1, wherein the multiviewautostereoscopic display is configured for providing views viewable by afirst viewer and a second viewer, wherein the at least one eye trackerincludes a first eye tracker for tracking eyes of the first viewer and asecond eye tracker for tracking eyes of the second viewer, and whereinthe processing system is further configured for: rendering specificviews based upon a first scene for each detected eye of the firstviewer, rendering additional views based upon a second scene for eachdetected eye of the second viewer, the second scene being at leastpartially different from the first scene, and generating controlinformation for the array of angular pixels such that, simultaneously,the specific views for each detected eye of the first viewer aredisplayed in viewing zones in which the eyes of the first viewer weredetected, and the additional views for each detected eye of the secondviewer are displayed in the viewing zones in which the second viewer'seyes were detected.
 14. The multiview autostereoscopic display of claim13, wherein the control information so generated by the processingsystem causes no view to be displayed outside the viewing zones in whichat least one of the first and second viewers' eyes were detected.
 15. Amultiview autostereoscopic display, comprising: a display area includingan array of angular pixels, where: each angular pixel is configured foremitting light that varies across a field of view of that angular pixel,the angular pixel including a plurality of emitters and optics, and atleast a portion of the plurality of emitters contribute to emitted lightwith a given intensity in a specific ray direction from the displayarea; memory configured for storing calibration information for thearray of angular pixels; and a processing system configured for:rendering a plurality of target views in specific viewing zones acrossthe field of view of the array of angular pixels, determining targetintensities for a set of ray directions from the display area based uponthe plurality of target views in the specific viewing zones in which theplurality of target views are to be generated, generating controlinformation for the plurality of emitters in the angular pixels in thedisplay area based upon the calibration information and the targetintensities, and generating a plurality of light field views using theangular pixels based upon the control information.
 16. The multiviewautostereoscopic display of claim 15, further comprising: at least oneeye tracker configured for determining locations of eyes of a viewer ofthe multiview autostereoscopic display and producing eye trackinginformation including the locations, wherein the processing system isfurther configured for selecting the set of ray directions at least inpart based upon the eye tracking information.
 17. The multiviewautostereoscopic display of claim 16, wherein generating controlinformation further includes identifying a subset of angular pixelsincluding emitters that contribute to a light field view at locations ofthe eyes of the viewer so detected based upon the eye trackinginformation.
 18. A multiview autostereoscopic display for providing aplurality of views to a viewer, comprising: a display area including anarray of angular pixels; an eye tracker; and a processing system,wherein each angular pixel is configured for emitting light that variesacross a field of view of that angular pixel, wherein the array ofangular pixels is configured for displaying different views in differentviewing zones across a field of view of the display, wherein the eyetracker is configured for detecting the presence of eyes of the viewerwithin specific viewing zones, and producing eye tracking informationrelated to locations of the eyes so detected within the specific viewingzones, wherein the processing system is configured for rendering aspecific view for each one of the eyes so detected within the viewingzone in which that eye was detected, and wherein the processing systemis further configured for generating control information for the arrayof angular pixels to cause the specific view for each one of the eyes sodetected to be displayed in the viewing zone in which that eye wasdetected.
 19. The multiview autostereoscopic display of claim 18,wherein the processing system is further configured for rendering atleast one of reduced resolution views, reduced density views, andtwo-dimensional views for display in specific viewing zones based uponthe eye tracking information.
 20. The multiview autostereoscopic displayof claim 18, wherein the control information includes activationinformation for the array of angular pixels to control activation ofemitters within the array of angular pixels based upon the eye trackinginformation to turn off a portion of the emitters contributing to viewsoutside of the viewing zone in which that eye was detected.
 21. Themultiview autostereoscopic display of claim 18, wherein the processingsystem is further configured for optimizing the control information formitigating at least one of aliasing, ghosting, optical crosstalk betweenviewing zones, and pixel-level optical crosstalk.